{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "afd55886-5f5b-4794-838e-ef8179fb0394",
   "metadata": {},
   "source": [
    "##### **** These pip installs need to be adapted to use the appropriate release level. Alternatively, The venv running the jupyter lab could be pre-configured with a requirement file that includes the right release. Example for transform developers working from git clone:\n",
    "```\n",
    "make venv \n",
    "source venv/bin/activate \n",
    "pip install jupyterlab\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "4c45c3c6-e4d7-4e61-8de6-32d61f2ce695",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture\n",
    "## This is here as a reference only\n",
    "# Users and application developers must use the right tag for the latest from pypi\n",
    "%pip install data-prep-toolkit\n",
    "%pip install 'data-prep-toolkit-transforms[lang_id]'\n",
    "%pip install pandas"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "407fd4e4-265d-4ec7-bbc9-b43158f5f1f3",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true
   },
   "source": [
    "##### **** Configure the transform parameters. The set of dictionary keys holding DocIDTransform configuration for values are as follows: \n",
    "| Key name  | Default  | Description |\n",
    "|------------|----------|--------------|\n",
    "| _lang_id_model_credential_ | _unset_ | specifies the credential you use to get model. This will be huggingface token. [Guide to get huggingface token](https://huggingface.co/docs/hub/security-tokens) |\n",
    "| _lang_id_model_kind_ | _unset_ | specifies what kind of model you want to use for language identification. Currently, only `fasttext` is available. |\n",
    "| _lang_id_model_url_ | _unset_ |  specifies url that model locates. For fasttext, this will be repo nme of the model, like `facebook/fasttext-language-identification` |\n",
    "| _lang_id_content_column_name_ | `contents` | specifies name of the column containing documents |\n",
    "| _lang_id_output_lang_column_name_ | `lang` | specifies name of the output column to hold predicted language code |\n",
    "| _lang_id_output_score_column_name_ | `score` | specifies name of the output column to hold score of prediction |"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ebf1f782-0e61-485c-8670-81066beb734c",
   "metadata": {},
   "source": [
    "##### ***** Import required classes and modules"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "9669273a-8fcc-4b40-9b20-8df658e2ab58",
   "metadata": {},
   "outputs": [],
   "source": [
    "from dpk_lang_id.transform_python import LangId"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7234563c-2924-4150-8a31-4aec98c1bf33",
   "metadata": {},
   "source": [
    "##### ***** Setup runtime parameters for this transform"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "badafb96-64d2-4bb8-9f3e-b23713fd5c3f",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "10:01:42 INFO - lang_id parameters are : {'model_credential': 'PUT YOUR OWN HUGGINGFACE CREDENTIAL', 'model_kind': 'fasttext', 'model_url': 'facebook/fasttext-language-identification', 'content_column_name': 'text', 'output_lang_column_name': 'lang', 'output_score_column_name': 'score'}\n",
      "10:01:42 INFO - pipeline id pipeline_id\n",
      "10:01:42 INFO - code location None\n",
      "10:01:42 INFO - data factory data_ is using local data access: input_folder - test-data/input output_folder - output\n",
      "10:01:42 INFO - data factory data_ max_files -1, n_sample -1\n",
      "10:01:42 INFO - data factory data_ Not using data sets, checkpointing False, max files -1, random samples -1, files to use ['.parquet'], files to checkpoint ['.parquet']\n",
      "10:01:42 INFO - orchestrator lang_id started at 2025-01-17 10:01:42\n",
      "10:01:42 INFO - Number of files is 3, source profile {'max_file_size': 0.3023223876953125, 'min_file_size': 0.037346839904785156, 'total_file_size': 0.4433746337890625}\n",
      "10:01:43 INFO - Completed 1 files (33.33%) in 0.009 min\n",
      "10:01:44 INFO - Completed 2 files (66.67%) in 0.011 min\n",
      "10:01:44 INFO - Completed 3 files (100.0%) in 0.013 min\n",
      "10:01:44 INFO - Done processing 3 files, waiting for flush() completion.\n",
      "10:01:44 INFO - done flushing in 0.0 sec\n",
      "10:01:44 INFO - Completed execution in 0.024 min, execution result 0\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "LangId(input_folder= \"test-data/input\",\n",
    "        output_folder= \"output\",\n",
    "        lang_id_model_credential= \"PUT YOUR OWN HUGGINGFACE CREDENTIAL\",\n",
    "        lang_id_model_kind= \"fasttext\",\n",
    "        lang_id_model_url= \"facebook/fasttext-language-identification\",\n",
    "        lang_id_content_column_name= \"text\").transform()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c3df5adf-4717-4a03-864d-9151cd3f134b",
   "metadata": {},
   "source": [
    "##### **** The specified folder will include the transformed parquet files."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "7276fe84-6512-4605-ab65-747351e13a7c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['output/test_03.parquet',\n",
       " 'output/test_02.parquet',\n",
       " 'output/metadata.json',\n",
       " 'output/test_01.parquet']"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import glob\n",
    "glob.glob(\"output/*\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "845a75cf-f4a9-467d-87fa-ccbac1c9beb8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>count()</th>\n",
       "      <th>lang</th>\n",
       "      <th>score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>- Notice of name-email change.doc</td>\n",
       "      <td>6</td>\n",
       "      <td>en</td>\n",
       "      <td>0.858</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>- Nov13ENAOnly.doc</td>\n",
       "      <td>2</td>\n",
       "      <td>de</td>\n",
       "      <td>0.264</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>- OHIO_C~1.XLS</td>\n",
       "      <td>2</td>\n",
       "      <td>de</td>\n",
       "      <td>0.603</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>- Oneok(5-30)final.doc</td>\n",
       "      <td>1</td>\n",
       "      <td>vi</td>\n",
       "      <td>0.152</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>- OpeningBrief.doc</td>\n",
       "      <td>6</td>\n",
       "      <td>ko-Hang</td>\n",
       "      <td>0.365</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>195</th>\n",
       "      <td>- invite.doc</td>\n",
       "      <td>2</td>\n",
       "      <td>ro</td>\n",
       "      <td>0.717</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>196</th>\n",
       "      <td>- issues wrt portland and calgary signing shor...</td>\n",
       "      <td>2</td>\n",
       "      <td>en</td>\n",
       "      <td>0.997</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>197</th>\n",
       "      <td>- jan3102.XLS</td>\n",
       "      <td>2</td>\n",
       "      <td>de</td>\n",
       "      <td>0.399</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>198</th>\n",
       "      <td>- job market.gif</td>\n",
       "      <td>2</td>\n",
       "      <td>en</td>\n",
       "      <td>0.791</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>199</th>\n",
       "      <td>- kick~1.mpe</td>\n",
       "      <td>4</td>\n",
       "      <td>eo</td>\n",
       "      <td>0.253</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>200 rows × 4 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                  text  count()     lang  \\\n",
       "0                    - Notice of name-email change.doc        6       en   \n",
       "1                                   - Nov13ENAOnly.doc        2       de   \n",
       "2                                       - OHIO_C~1.XLS        2       de   \n",
       "3                               - Oneok(5-30)final.doc        1       vi   \n",
       "4                                   - OpeningBrief.doc        6  ko-Hang   \n",
       "..                                                 ...      ...      ...   \n",
       "195                                       - invite.doc        2       ro   \n",
       "196  - issues wrt portland and calgary signing shor...        2       en   \n",
       "197                                      - jan3102.XLS        2       de   \n",
       "198                                   - job market.gif        2       en   \n",
       "199                                       - kick~1.mpe        4       eo   \n",
       "\n",
       "     score  \n",
       "0    0.858  \n",
       "1    0.264  \n",
       "2    0.603  \n",
       "3    0.152  \n",
       "4    0.365  \n",
       "..     ...  \n",
       "195  0.717  \n",
       "196  0.997  \n",
       "197  0.399  \n",
       "198  0.791  \n",
       "199  0.253  \n",
       "\n",
       "[200 rows x 4 columns]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "pd.read_parquet('output/test_01.parquet', engine='pyarrow')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7aef6ac9-96cf-40ad-a472-b5d9036436e5",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
